Systems and methods of multiview style transfer

ABSTRACT

A system and method of multiview style transfer apply a style transfer to individual views of a multiview image in a way that produces consistent results across all images. In some embodiments, the multiview style transfer includes receiving first and second images representative of first and second perspectives of a scene and first and second disparity maps corresponding to the first and second images, generating a first stylized image, generating a stylized shifted image based on the first stylized image and the first disparity map, generating a second stylized image based on a guided filter of the stylized shifted image and the second image, and generating a first and second stylized image based on the stylized shifted images and the disparity maps.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation patent application of and claimspriority to International Application No. PCT/US2021/015570, filed Jan.28, 2021, which claims the benefit of priority to U.S. ProvisionalPatent Application Ser. No. 62/983,739, filed Mar. 1, 2020, both ofwhich are incorporated by reference in their entirety herein.

BACKGROUND

Electronic displays are a nearly ubiquitous medium for communicatinginformation to users of a wide variety of devices and products. Mostcommonly employed electronic displays include the cathode ray tube(CRT), plasma display panels (PDP), liquid crystal displays (LCD),electroluminescent displays (EL), organic light emitting diode (OLED)and active matrix OLEDs (AMOLED) displays, electrophoretic displays (EP)and various displays that employ electromechanical or electrofluidiclight modulation (e.g., digital micromirror devices, electrowettingdisplays, etc.). Generally, electronic displays may be categorized aseither active displays (i.e., displays that emit light) or passivedisplays (i.e., displays that modulate light provided by anothersource). Among the most obvious examples of active displays are CRTs,PDPs and OLEDs/AMOLEDs.

Content and related information displayed on electronic displays isgenerally rendered using a graphics processing unit (GPU) and the passedto a display driver. In some cases, especially images or photographs,original content may be manipulated or modified prior to rendering.Manipulation or modification may be provided using the GPU or anotherprocessor associated with displaying the information, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features of examples and embodiments in accordance with theprinciples described herein may be more readily understood withreference to the following detailed description taken in conjunctionwith the accompanying drawings, where like reference numerals designatelike structural elements, and in which:

FIG. 1A illustrates a perspective view of a multiview display in anexample, according to an embodiment consistent with the principlesdescribed herein.

FIG. 1B illustrates a graphical representation of angular components ofa light beam having a particular principal angular directioncorresponding to a view direction of a multiview display in an example,according to an embodiment consistent with the principles describedherein.

FIG. 2 illustrates a flowchart of a multiview style transfer method inan example, according to an embodiment consistent with the principlesdescribed herein.

FIG. 3 illustrates a guided filter application in an example, accordingto an embodiment consistent with the principles described herein.

FIG. 4 illustrates a flowchart of a multiview style transfer outputarray in an example, according to an embodiment consistent with theprinciples described herein.

FIGS. 5A-5C illustrate multiview style transfer method results in anexample, according to an embodiment consistent with the principlesdescribed herein.

FIGS. 6A-6B illustrate a multiview style transfer device in variousexamples, according to an embodiment consistent with the principlesdescribed herein.

Certain examples and embodiments have other features that are one of inaddition to and in lieu of the features illustrated in theabove-referenced figures. These and other features are detailed belowwith reference to the above-referenced figures.

DETAILED DESCRIPTION

Examples in accordance with the principles described herein providemultiview style transfer with application to electronic display. Inparticular, embodiments of the principles described herein may provide amethod of applying style transfer to individual views of a multiviewimage in a way that produces consistent results across all images. Insome embodiments, the multiview style transfer may run with graphicsprocessing unit (GPU) acceleration on a mobile device. According tovarious embodiments, multiview style transfer includes a modularpipeline where individual algorithms can be de-coupled from the system.

In some embodiments, a neural network may be employed to provide neuralstyle transfer to multiview images having more than two viewpoints.According to various embodiments, the multiview style transfer describedherein may result in style transfer that is substantially consistentbetween all views, that uses stereo information from both of a pair ofviews, and that may be GPU-accelerated for on-demand performance onmobile platforms.

According to various embodiments, backlighting of an electronic displaymay employ a multibeam diffraction grating to diffractively couple light(e.g., of different colors), or more generally may employ a multibeamelement to scatter light out of a light guide and to direct thecoupled-out, scattered-out or emitted light in different principleangular directions that correspond to a viewing direction or a pluralityof viewing directions of an electronic display. In some examples orembodiments, the light beams having the different principal angulardirections (also referred to as ‘the differently directed light beams’)and, in some embodiments having the different colors, may be employed todisplay three-dimensional (3D) information as a multiview image. Forexample, the differently directed, different color light beams may bemodulated and serve as pixels of a ‘glasses free’ 3D or multiviewelectronic display.

Herein a ‘two-dimensional display’ or ‘2D display’ is defined as adisplay configured to provide a view of an image that is substantiallythe same regardless of a direction from which the image is viewed (i.e.,within a predefined viewing angle or range of the 2D display). Aconventional liquid crystal display (LCD) found in many smart phones andcomputer monitors are examples of 2D displays. In contrast herein, a‘multiview display’ is defined as an electronic display or displaysystem configured to provide different views of a multiview image in orfrom different view directions. In particular, the different views mayrepresent different perspective views of a scene or object of themultiview image. Uses of unilateral backlighting and unilateralmultiview displays described herein include, but are not limited to,mobile telephones (e.g., smart phones), watches, tablet computers,mobile computers (e.g., laptop computers), personal computers andcomputer monitors, automobile display consoles, cameras displays, andvarious other mobile as well as substantially non-mobile displayapplications and devices.

FIG. 1A illustrates a perspective view of a multiview display 110 in anexample, according to an embodiment consistent with the principlesdescribed herein. As illustrated in FIG. 1A, the multiview display 100comprises a screen 112 to display a multiview image to be viewed. Thescreen 112 may be a display screen of a telephone (e.g., mobiletelephone, smart phone, etc.), a tablet computer, a laptop computer, acomputer monitor of a desktop computer, a camera display, or anelectronic display of substantially any other device, for example.

The multiview display 100 provides different views 114 of the multiviewimage in different view directions 116 relative to the screen 112. Theview directions 116 are illustrated as arrows extending from the screen112 in various different principal angular directions; the differentviews 114 are illustrated as shaded polygonal boxes at the terminationof the arrows (i.e., depicting the view directions 116); and only fourviews 114 and four view directions 116 are illustrated, all by way ofexample and not limitation. Note that while the different views 114 areillustrated in FIG. 1A as being above the screen, the views 114 actuallyappear on or in a vicinity of the screen 112 when the multiview image isdisplayed on the multiview display 110. Depicting the views 114 abovethe screen 112 is only for simplicity of illustration and is meant torepresent viewing the multiview display 110 from a respective one of theview directions 116 corresponding to a particular view 114. A 2D displaymay be substantially similar to the multiview display 110, except thatthe 2D Display is generally configured to provide a single view (e.g.,one view of the multiple different views 114) of a displayed image asopposed to the different views 114 of the multiview image provided bythe multiview display 100.

A view direction or equivalently a light beam having a directioncorresponding to a view direction of a multiview display generally has aprincipal angular direction given by angular components {θ, ϕ}, bydefinition herein. The angular component θ is referred to herein as the‘elevation component’ or ‘elevation angle’ of the light beam. Theangular component ϕ is referred to as the ‘azimuth component’ or‘azimuth angle’ of the light beam. By definition, the elevation angle θis an angle in a vertical plane (e.g., perpendicular to a plane of themultiview display screen) while the azimuth angle ϕ is an angle in ahorizontal plane (e.g., parallel to the multiview display screen plane).

FIG. 1B illustrates a graphical representation of the angular components{θ, ϕ} of a light beam 120 having a particular principal angulardirection corresponding to a view direction (e.g., view direction 116 inFIG. 1A) of a multiview display in an example, according to anembodiment consistent with the principles described herein. In addition,the light beam 120 is emitted or emanates from a particular point, bydefinition herein. That is, by definition, the light beam 120 has acentral ray associated with a particular point of origin within themultiview display. FIG. 1B also illustrates the light beam (or viewdirection) point of origin O.

Herein, a ‘diffraction grating’ is generally defined as a plurality offeatures (i.e., diffractive features) arranged to provide diffraction oflight incident on the diffraction grating. In some examples, theplurality of features may be arranged in a periodic or quasi-periodicmanner. For example, the diffraction grating may include a plurality offeatures (e.g., a plurality of grooves or ridges in a material surface)arranged in a one-dimensional (1D) array. In other examples, thediffraction grating may be a two-dimensional (2D) array of features. Thediffraction grating may be a 2D array of bumps on or holes in a materialsurface, for example.

As such, and by definition herein, the ‘diffraction grating’ is astructure that provides diffraction of light incident on the diffractiongrating. If the light is incident on the diffraction grating from alight guide, the provided diffraction or diffractive scattering mayresult in, and thus be referred to as, ‘diffractive coupling’ in thatthe diffraction grating may couple light out of the light guide bydiffraction. The diffraction grating also redirects or changes an angleof the light by diffraction (i.e., at a diffractive angle). Inparticular, as a result of diffraction, light leaving the diffractiongrating generally has a different propagation direction than apropagation direction of the light incident on the diffraction grating(i.e., incident light). The change in the propagation direction of thelight by diffraction is referred to as ‘diffractive redirection’ herein.Hence, the diffraction grating may be understood to be a structureincluding diffractive features that diffractively redirects lightincident on the diffraction grating and, if the light is incident from alight guide, the diffraction grating may also diffractively couple outthe light from the light guide.

Further, by definition herein, the features of a diffraction grating arereferred to as ‘diffractive features’ and may be one or more of at, inand on a material surface (i.e., a boundary between two materials). Thesurface may be a surface of a light guide, for example. The diffractivefeatures may include any of a variety of structures that diffract lightincluding, but not limited to, one or more of grooves, ridges, holes andbumps at, in or on the surface. For example, the diffraction grating mayinclude a plurality of substantially parallel grooves in the materialsurface. In another example, the diffraction grating may include aplurality of parallel ridges rising out of the material surface. Thediffractive features (e.g., grooves, ridges, holes, bumps, etc.) mayhave any of a variety of cross sectional shapes or profiles that providediffraction including, but not limited to, one or more of a sinusoidalprofile, a rectangular profile (e.g., a binary diffraction grating), atriangular profile and a saw tooth profile (e.g., a blazed grating).

According to various examples described herein, a diffraction grating(e.g., a diffraction grating of a multibeam element, as described below)may be employed to diffractively scatter or couple light out of a lightguide (e.g., a plate light guide) as a light beam. In particular, adiffraction angle θ_(m) of or provided by a locally periodic diffractiongrating may be given by equation (1) as:

$\begin{matrix}{\theta_{m} = {\sin^{- 1}\left( {{n\sin\theta_{i}} - \frac{m\lambda}{d}} \right)}} & (1)\end{matrix}$

where λ is a wavelength of the light, m is a diffraction order, n is anindex of refraction of a light guide, d is a distance or spacing betweenfeatures of the diffraction grating, θ_(i) is an angle of incidence oflight on the diffraction grating. For simplicity, equation (1) assumesthat the diffraction grating is adjacent to a surface of the light guideand a refractive index of a material outside of the light guide is equalto one (i.e., n_(out)=1). In general, the diffraction order m is givenby an integer. A diffraction angle θ_(m) of a light beam produced by thediffraction grating may be given by equation (1) where the diffractionorder is positive (e.g., m>0). For example, first-order diffraction isprovided when the diffraction order m is equal to one (i.e., m=1).

Herein, the term ‘multiview’ or equivalently ‘multi-view’ as used in theterms ‘multiview image’ and ‘multiview display’ is defined as aplurality of views representing different perspectives or includingangular disparity between views of the view plurality. In addition,herein the term ‘multiview’ explicitly includes more than two differentviews (i.e., a minimum of three views and generally more than threeviews), by definition herein. As such, ‘multiview display’ as employedherein is explicitly distinguished from a stereoscopic display thatincludes only two different views to represent a scene or an image. Notehowever, while multiview images and multiview displays may include morethan two views, by definition herein, multiview images may be viewed(e.g., on a multiview display) as a stereoscopic pair of images byselecting only two of the multiview views to view at a time (e.g., oneview per eye).

A ‘multiview pixel’ is defined herein as a set of sub-pixelsrepresenting ‘view’ pixels in each of a similar plurality of differentviews of a multiview display. In particular, a multiview pixel may havean individual sub-pixel corresponding to or representing a view pixel ineach of the different views of the multiview image. Moreover, thesub-pixels of the multiview pixel are so-called ‘directional pixels’ inthat each of the sub-pixels is associated with a predetermined viewdirection of a corresponding one of the different views, by definitionherein. Further, according to various examples and embodiments, thedifferent view pixels represented by the subpixels of a multiview pixelmay have equivalent or at least substantially similar locations orcoordinates in each of the different views. For example, a firstmultiview pixel may have individual sub-pixels corresponding to viewpixels located at {x₁, y₁} in each of the different views of a multiviewimage, while a second multiview pixel may have individual sub-pixelscorresponding to view pixels located at {x₂, y₂} in each of thedifferent views, and so on.

Herein, a ‘multiview image’ is defined as a plurality of images (i.e.,greater than three images) wherein each image of the pluralityrepresents a different view corresponding to a different view directionof the multiview image. As such, the multiview image is a collection ofimages (e.g., two-dimensional images) which, when display on a multiviewdisplay, may facilitate a perception of depth and thus appear to be animage of a 3D scene to a viewer, for example.

Embodiments consistent with the principles described herein may beimplemented using a variety of devices and circuits including, but notlimited to, one or more of integrated circuits (ICs), very large scaleintegrated (VLSI) circuits, application specific integrated circuits(ASIC), field programmable gate arrays (FPGAs), digital signalprocessors (DSPs), graphical processor unit (GPU), and the like,firmware, software (such as a program module or a set of instructions),and a combination of two or more of the above. For example, anembodiment or elements thereof may be implemented as circuit elementswithin an ASIC or a VLSI circuit. Implementations that employ an ASIC ora VLSI circuit are examples of hardware-based circuit implementations.

In another example, an embodiment may be implemented as software using acomputer programming language (e.g., C/C++) that is executed in anoperating environment or a software-based modeling environment (e.g.,MATLAB®, MathWorks, Inc., Natick, Mass.) that is further executed by acomputer (e.g., stored in memory and executed by a processor or agraphics processor of a general purpose computer). Note that one or morecomputer programs or software may constitute a computer-programmechanism, and the programming language may be compiled or interpreted,e.g., configurable or configured (which may be used interchangeably inthis discussion), to be executed by a processor or a graphics processorof a computer.

In yet another example, a block, a module or an element of an apparatus,device or system (e.g., image processor, camera, etc.) described hereinmay be implemented using actual or physical circuitry (e.g., as an IC oran ASIC), while another block, module or element may be implemented insoftware or firmware. In particular, according to the definitionsherein, some embodiments may be implemented using a substantiallyhardware-based circuit approach or device (e.g., ICs, VLSI, ASIC, FPGA,DSP, firmware, etc.), while other embodiments may also be implemented assoftware or firmware using a computer processor or a graphics processorto execute the software, or as a combination of software or firmware andhardware-based circuitry, for example.

Further, as used herein, the article ‘a’ is intended to have itsordinary meaning in the patent arts, namely ‘one or more’. For example,‘a multiview display’ means one or more multiview display and as such,‘the multiview display’ means ‘the multiview display(s)’ herein. Also,any reference herein to ‘top,’ ‘bottom,’ ‘upper,’ ‘lower,’ ‘up,’ ‘down,’‘front,’ ‘back,’ ‘first,’ ‘second,’ ‘left’ or ‘right’ is not intended tobe a limitation herein. Herein, the term ‘about’ when applied to a valuegenerally means within the tolerance range of the equipment used toproduce the value, or may mean plus or minus 10%, or plus or minus 5%,or plus or minus 1%, unless otherwise expressly specified. Further, theterm ‘substantially’ as used herein means a majority, or almost all, orall, or an amount within a range of about 51% to about 100%. Moreover,examples herein are intended to be illustrative only and are presentedfor discussion purposes and not by way of limitation.

FIG. 2 illustrates a flowchart of a multiview style transfer method 200in an example, according to an embodiment consistent with the principlesdescribed herein. As illustrated in FIG. 2 , an example method 200 forstylizing multiview images includes receiving style information andinput image information and synthesizing, or providing, multiple outputimages consistent with the received style information. For example, themethod 200 includes receiving a style guide G_(s) 205, left disparitymap Δ_(l) 210, right disparity map Δ_(r) 215, left stereo input viewI_(l) 220, and right stereo input view I_(r) 225. The method 200 isdescribed with respect to left and right images, however a correspondingmethod may be applied to right and left images, respectively.

In an example, left stereo input view I_(l) 220 and right stereo inputview I_(r) 225 may be provided with corresponding left disparity mapΔ_(l) 210 and right disparity map Δ_(r) 215. In another example, thedisparity maps 210 and 215 may be estimated based on left stereo inputview I_(l) 220 and right stereo input view I_(r) 225. In an example,disparity maps 210 and 215 are estimated based on a neural networktrained on a plurality of input stereo images.

In the example of FIG. 2 , method 200 includes applying a style transfernetwork to stylize 230 left stereo input view I_(l) 220 based on styleguide G_(s) 205, which generates a stylized left image S_(l) 240. Theapplication of the style transfer network to stylize 230 only one of theinput stereo views provides various advantages. Stylization 230 of oneof the two stereo images provides stylistic consistency between multipleviews, which reduces or eliminates effects caused by the sensitivity ofstyle transfer networks to small changes in the input.

For example, a problem in stylizing each view individually can include aparallax effect between stereo views, such as may cause visualdifferences, or may result in stylized features changing location orappearance between stylized stereo images, such as shown in FIG. 4enlarged images 450. Such inconsistencies can cause viewer fatigue, forexample, when viewing 3D content on a multiview display. A solution tothe problem can include using stylization 230 of one of the two stereoimages. The solution can further provide a reduced computational cost,such as for multiview display devices on portable electronic devices(e.g., smartphones). That is, stylizing once and synthesizing manyoutput frames or output views is considerably more computationallyefficient than stylizing every output frame or output view individually.Another advantage of the present solution can include that thestylization 230 of one of the two stereo images can provide improvedcompatibility with various style transfer networks. For example, thesolution can help reduce or eliminate modifications and retrainingspecifically for multiview rendering, and thereby method 200 may provideimproved functionality with various style transfer networks, and mayprovide improved quality and performance.

The example of method 200 includes a re-projection 235 of first stylizedleft image S_(l) 240 based on left disparity map Δ_(l) 210 to generatestylized right image S_(r) 245. In an example, stylized left image S_(l)240 is re-projected to generate stylized right image S_(r) 245 at thesame viewpoint as right stereo input view I_(r) 225, such as using aview synthesis module. In an example, the re-projection 235 may beperformed on one or more of a central processing unit (CPU) and agraphical processing unit (GPU). In an example, re-projection 235includes one or more of forward warping, a depth test, and anin-painting technique to sample nearby regions such as to fillde-occluded regions. Forward warping is an image distortion process thatapplies a transformation to a source image. Pixels from the source imagemay be processed in a scanline order and the results are projected ontoa target image. A depth test is a process where fragments of an imagethat are processed or to be processed by a shader have depth values thatare tested with respect to a depth of a sample to which it is beingwritten. Fragments are discarded when the test fails. And a depth bufferis updated with the fragment's output depth when the test passes.In-painting refers to filling in missing or unknown regions of an image.Some techniques involve predicting pixel vales based on nearby pixels orreflecting nearby pixels onto an unknown or missing region. Missing orunknown regions of an image may result from scene de-occlusion, whichrefers to a scene object that is partially covered by another sceneobject. In this respect, re-projection may involve image processingtechniques to construct a new perspective of a scene from an originalperspective. The resultant generated stylized right image S_(r) 245 isthus a re-projection of the stylized left view to the right viewpoint,where the style features of stylized right image S_(r) 245 aretransported to their corresponding positions in the generated stylizedright image S_(r) 245.

The example of method 200 includes applying guided filter module 250 tostylized left image S_(l) 240 to generate left filtered stylized viewS_(l)′ 260, and similarly applying guided filter module 255 to stylizedright image S_(r) 245 to generate right filtered stylized view S_(r)′265. In an example, the guided filter module 250 includes a filterconfigured for refining stylized left image S_(l) 240 and stylized rightimage S_(r) 245 using edge-aware guided filtering. The edge-aware guidedfiltering can be based on detected edges in left stereo input view I_(l)220 and right stereo input view I_(r) 225. For example, when viewingimages on a multiview display, a quality of edge placements can enhanceor detract from the 3D perception experience, however the style transferprocess can, in some examples, degrade the edges of objects in the 3Dscene. By applying guided filter module 250 to the stylized images 240and 245 using their corresponding un-stylized views 220 and 225 asguides, the edges of original 3D objects can be reinforced whilereducing the stylization of the edges, thus resulting in a moreimmersive or robust 3D experience. An example of the application ofguided filter module 250 is shown in FIG. 3 , described below.

The method 200 includes synthesizing 270 left filtered stylized viewS_(l)′ 260 and right filtered stylized view S_(r)′ 265 to generatemultiple stylized images S₁, S₂, S₃, and S₄ 280 corresponding torespective different viewpoints. In an example, synthesizing 270includes re-projecting left filtered stylized view S_(l)′ 260 and rightfiltered stylized view S_(r)′ 265 to multiple viewpoints x₁, x₂, . . . ,x_(n), such as can be based on left disparity map Δ_(l) 210 and rightdisparity map Δ_(r) 215. This re-projection is similar to re-projection235 as applied to multiple viewpoints x₁, x₂, . . . , x_(n). In anexample, each of the viewpoint stylized images S₁, S₂, S₃, and S₄ 280 isbased on a re-projection of left filtered stylized view S_(l)′ 260 andright filtered stylized view S_(r)′ 265 to each of multiple viewpointsx₁, x₂, . . . , x_(n), and blending based on proximity of the viewpointto the left and right viewpoints corresponding with left stereo inputview I_(l) 220 and right stereo input view I_(r) 225.

FIG. 3 illustrates a guided filter application 300 in an example,according to an embodiment consistent with the principles describedherein. FIG. 3 shows an example of a stylized image S before applicationof a guided filter 310 and after application of the guided filter 320.Enlarged pre-filter image 315 and enlarged post-filter image 325 showthat applying a guided filter increases the consistency with the edgesin the original 3D scene while reducing the effect of edges introducedby style transfer.

FIG. 4 illustrates a flowchart of a multiview style transfer outputarray 400 in an example, according to an embodiment consistent with theprinciples described herein. As illustrated in FIG. 4 , a style guide405 is applied to a left disparity map 410, a right disparity map 415, aleft stereo input view 420, and a right stereo input view 425. A firstexample stylized image pair 430 and 435 corresponds to images that havebeen stylized and projected, but without application of a guided edgefilter. As can be seen in images 450 enlarged from the first examplestylized image pair 430 and 435, inconsistent features and otherinconsistent styling may result from a pair of stereo images that havebeen stylized and projected without application of a guided edge filter.These inconsistencies may reduce the viewer's ability to focus onstereoscopic images, thereby increasing 3D viewing fatigue. A secondexample stylized image pair 460 and 465 corresponds to images that havebeen stylized, projected, and filtered using a guided edge filter. Ascan be seen in images 470 enlarged from the second example stylizedimage pair 460 and 465, the consistency of features and styling isimproved by the application of a guided edge filter. This improvedconsistency improves user focus and comfort in viewing multiple imageson a multiview display.

The present systems and methods for a multiview style transfer MSTprovide various advantages over other solutions. Table 1 shows a meanruntime comparison in some embodiments for rendering a stylizedmultiview image with 4 views (e.g., viewpoints), 8 views, and 16 views,using each of six different methods or techniques:

TABLE 1 Multiview Style Transfer Output Runtime Comparison Time taken(ms) Method 4 Views 8 Views 16 Views Baseline CPU 8352 16682 33358Baseline GPU 1405 2832 5768 Approach A 843 849 858 Approach B 746 9951213 MST CPU 2311 2394 2576 MST GPU 561 567 576

Table 1 compares baseline CPU and GPU solutions, Approaches A and B, andthe present multiview style transfer (MST) CPU and GPU solutions. Thebaseline CPU and baseline GPU solutions naively apply neural styletransfer to each of the synthesized views individually. As shown inTable 1, the present MST CPU and MST GPU scale linearly with a number ofperspective viewpoints. This provides computational efficiencyimprovements over baseline CPU and baseline GPU solutions, which do notscale linearly. The present MST CPU and MST GPU further producestyle-consistent views and thus improved computational efficiency.

The present MST CPU and MST GPU provide improvements over Approach A.Approach A includes applying a neural style-transfer to each of thestereoscopic input views, then performing a novel view synthesis usingthe stylized pair and the original disparity maps as inputs. WhileApproach A runs faster than baseline CPU and baseline GPU, the renderedviews produce undesirable ghosting artifacts and an overall inconsistentstyling between the stereoscopic pair, which can lead to viewingfatigue.

Approach B seeks to improve the style inconsistency of the output imagesover Approach A. Approach B includes applying a neural style only to theinput left image to create the stylized left image and then synthesizingnovel views only from this stylized left image. Approach B furtherincludes performing view synthesis simultaneously using both theoriginal naturalistic left and right images, where this naturalisticmultiview image is used as a guide for a guided filter pass on thestylized multiview image. The resulting multiview image is sharpened toreduce blurring artifacts. This method produces consistently styledviews with relatively sharp edges, however Approach B limits the deptheffect due to using only the left image for the styled novel viewsynthesis. Additionally, Approach B results in a reduction of lined-upedges in the guided filtering step, and ghosting artifacts are producedaround the edges in the output views.

The multiview style transfer method 200 provides improvements overbaseline, Approach A, and Approach B, while providing improvedcomputational efficiency. In particular, multiview style transfer method200 provides improved multiview consistent stylized images withon-demand performance, including when GPU-accelerated on mobile devices.

FIGS. 5A-5C illustrate multiview style transfer method results 500 invarious examples, according to an embodiment consistent with theprinciples described herein. FIG. 5A illustrates a 4-view example thattakes inputs 510 such as original stereo pairs, disparity maps, andstyle guide, and generates outputs 515. Similarly, FIG. 5B illustratesan 8-view example that takes inputs 520 and generates outputs 525, andFIG. 5C illustrates a 16-view example that takes inputs 530 andgenerates outputs 535. As can be seen in FIGS. 5A-5C, the presentmultiview style transfer method 200 results in consistent stylization ofobjects within each synthesized image regardless of object position,rotation, or occlusion. The different views in the examples of FIGS.5A-5C can correspond to respective different views or view perspectivesin a multiview display.

FIGS. 6A-6B illustrate a multiview style transfer device 600 in variousexamples, according to an embodiment consistent with the principlesdescribed herein. FIG. 6A illustrates a block diagram of an electronicdevice 600 that includes multiview display 628 in an example, accordingto an embodiment of the principles described herein. As illustrated, theelectronic device 600 comprises a graphics processing unit (GPU) 610.The graphics processing unit 610 is configured to generate a stylizedmultiview image 612 with separate 3D viewpoints (such as the stylizedmultiview image described previously).

After receiving the stylized multiview image 612, a driver 616 may storethe stylized multiview image 612 in a buffer 618. Note that the buffer618 may be able to store the entire stylized multiview image 612 withthe 3D views, such as a full frame of 3D video. Then, a mapping circuit620 (such as control or routing logic, and more generally a mapping or atransformation block) transforms the stylized multiview image 612 into acomposite image 622. Next, a driver circuit 624 drives or applies pixeldrive signals 626 to the multiview display 628 based on the compositeimage 622. In some embodiments, the stylized multiview image 612 has oris compatible with an image file having one of multiple differentformats.

Instead of a separate driver 616, in some embodiments some or all of thefunctionality in the driver 616 is included in the graphics processingunit. This is shown in FIG. 6B, which illustrates a block diagram of anelectronic device 630 that includes the multiview display 628 in anexample, according to another embodiment of the principles describedherein. In particular, in FIG. 6B, a graphics processing unit 610includes components of the driver 616.

While FIGS. 6A and 6B illustrate the image-processing technique inelectronic devices that include the multiview display 628, in someembodiments the image-processing technique is implemented in one or morecomponents in one of the electronic devices 600 and 630, such as one ormore components in the multiview display 628, which may be provideseparately from or in conjunction with a remainder of the multiviewdisplay 628 or one of the electronic devices 600 and 630.

In some embodiments, electronic devices 600 and 630 may includeprocessing circuitry such as, for example, a central processing unit(CPU) that is configured to execute instructions stored in memory. Theinstructions may be part of an application that is supported by anoperating system. The instructions may be part of a software programroutine that is executed by a CPU, GPU, or a combination thereof. Forexample, view synthesis may be implemented as software executed by a CPUor GPU while the guided filter is implemented as software that isexecuted by the CPU.

Various aspects of the present disclosure can help provide a solution tothe stylization problems identified herein. For instance, Example 1 caninclude a multiview style transfer system comprising: processingcircuitry; and a memory that includes, instructions, the instructions,when executed by the processing circuitry, cause the processingcircuitry to: receive a first image representative of a firstperspective of a scene, a first disparity map corresponding to the firstimage, a second image representative of a second perspective of thescene, and a second disparity map corresponding to the second image;generate a first stylized image, the first stylized image representativeof a style transfer model applied to the first image; generate astylized shifted image based on the first stylized image and the firstdisparity map, the stylized shifted image including a first shift of thefirst stylized image to the second perspective; and generate a secondstylized image based on a guided filter of the stylized shifted imageand the second image, the guided filter to process edge characteristicsin the second stylized image based on the second image. A shifted imageor shift image is an image that has a perspective view that is shiftedfrom an original image. Shifting refers to shifting a view of an imageto generate a new image. Shifting may be implemented using are-projection technique. To generate a shift image from an originalimage, various regions of the original image may be stretched,relocated, or warped.

In Example 2, the subject matter of Example 1 includes, a multiviewdisplay, wherein the instructions further cause the processing circuitryto: generate a first stylized image based on the stylized shifted imageand the first disparity map; generate a second stylized image based onthe second stylized image and the second disparity map, wherein thefirst stylized image and the second stylized image are configured forconcurrent use by the multiview display; and display the first stylizedimage and the second stylized image on the multiview display.

In Example 3, the subject matter of Example 2 includes, the instructionsfurther causing the processing circuitry to: synthesize a plurality ofstylized perspective views based on first stylized image and the secondstylized image; and display the plurality of stylized perspective viewson the multiview display.

In Example 4, the subject matter of Examples 1-3 includes, theinstructions further causing the processing circuitry to generate asecond filtered image based on application of a second guided filter tothe second stylized image, wherein the generation of the second stylizedimage is further based on the second filtered image.

In Example 5, the subject matter of Example 4 includes, the instructionsfurther causing the processing circuitry to generate a first filteredimage based on application of a first guided filter to the firststylized image, wherein the generation of the first stylized image isfurther based on the first filtered image.

In Example 6, the subject matter of Example 5 includes, wherein: thefirst guided filter includes a first guided sharpening filter, the firstguided sharpening filter to sharpen the first stylized image based on afirst plurality of edges within the first image; and the second guidedfilter includes a second guided sharpening filter, the second guidedsharpening filter to sharpen the second stylized image based on a secondplurality of edges within the second image. In some embodiments, aguided sharpening filter may be used to sharpen detected edges by usinga high pass filter that remove low frequencies.

In Example 7, the subject matter of Examples 1-6 includes, theinstructions further causing the processing circuitry to generate thefirst stylized image based on the first image and the style transfermodel.

In Example 8, the subject matter of Example 7 includes, wherein thestyle transfer model includes a style transfer neural network trainedvia machine learning on a plurality of target style images.

In Example 9, the subject matter of Examples 7-8 includes, theinstructions further causing the processing circuitry to: receive atarget style image; and identify a target style based on application ofa neural style transfer (NST) algorithm to the target style image; andgenerate the style transfer model based on the target style.

In Example 10, the subject matter of Examples 1-9 includes, theinstructions further causing the processing circuitry to: generate thefirst disparity map based on the first image and the second image, thefirst disparity map representing differences in horizontal coordinatesof a first plurality of image points in the first image relative to asecond plurality of image points in the second image; and generate thesecond disparity map based on the first image and the second image, thesecond disparity map representing differences in horizontal coordinatesof the second plurality of image points in the second image relative tothe first plurality of image points in the first image. For example,disparity map indicates an apparent pixel difference between views ofmultiview image. In this respect, a disparity map controls the apparentdisparity of rendered pixels by specifying where pixels should berendered on the multiview display. When disparity is about zero, thepixels representing an object appear to the viewer at the same locationacross different views. When rendered on a multiview display, pixelshaving about zero disparity appear to viewer as located on the screendisplay while pixels having non-zero disparity appear either in front ofor behind the screen of the display.

The differences in horizontal coordinates across different views resultin differences in pixel locations of the same object that is viewed fromdifferent perspectives giving rise to disparity. In some embodiments, adisparity map may indicate vertical disparity, horizontal disparity, orboth. Thus, the difference between corresponding pixels of differentviews may be in either the vertical direction, horizontal direction, orboth.

In Example 11, the subject matter of Examples 1-10 includes, theinstructions further causing the processing circuitry to: generate afirst extrapolated image based on the first stylized image and the firstdisparity map, the first extrapolated image representing a firstsynthesized viewpoint extrapolated to a third viewpoint based on thefirst perspective of the scene associated with the first image; generatea second extrapolated image based on the second stylized image and thesecond disparity map, the second extrapolated image representing asecond synthesized viewpoint extrapolated to a fourth viewpoint based onthe second perspective of the scene associated with the second image;and display the first extrapolated image and the second extrapolatedimage on the multiview display. View synthesis may involve artificiallypredicting, extrapolating, or interpolating new views from one or moreoriginal views using computer vision techniques, forward warping, adepth test, in-painting techniques or any combination thereof.

In Example 12, the subject matter of Examples 1-11 includes, theinstructions further causing the processing circuitry to: generate afirst stylized viewpoint image based on re-projecting the first stylizedimage from a first desired output viewpoint; generate a second stylizedviewpoint image based on re-projecting the second stylized image from asecond desired output viewpoint; and display the first stylizedviewpoint image and the second stylized viewpoint image on the multiviewdisplay. The desired output viewpoint corresponds to a principle angulardirection of a view that is produced by the multiview display.

In Example 13, the subject matter of Example 12 includes, wherein: thefirst desired output viewpoint is based on a first device viewpointassociated with a device multiview display; and the second desiredoutput viewpoint is based on a second device viewpoint associated with adevice multiview display.

In Example 14, the subject matter of Examples 12-13 includes, theinstructions further causing the processing circuitry to: generate athird stylized viewpoint image based on re-projecting the third stylizedimage from a third desired output viewpoint; generate a fourth stylizedviewpoint image based on re-projecting the fourth stylized image from afourth desired output viewpoint; and display the third stylizedviewpoint image and the fourth stylized viewpoint image on the multiviewdisplay.

Example 15 is a multiview style transfer method comprising: receiving afirst image representative of a first perspective of a scene, a firstdisparity map corresponding to the first image, a second imagerepresentative of a second perspective of the scene, and a seconddisparity map corresponding to the second image; generating a firststylized image, the first stylized image representative of a styletransfer model applied to the first image; generating a stylized shiftedimage based on the first stylized image and the first disparity map, thestylized shifted image including a first shift of the first stylizedimage to the second perspective; and generating a second stylized imagebased on a guided filter of the stylized shifted image and the secondimage, the guided filter to process edge characteristics in the secondstylized image based on the second image.

In Example 16, the subject matter of Example 15 includes, generating afirst stylized image based on the stylized shifted image and the firstdisparity map; and generating a second stylized image based on thesecond stylized image and the second disparity map, wherein the firststylized image and the second stylized image are configured forconcurrent use by a multiview display.

In Example 17, the subject matter of Example 16 includes, synthesizing aplurality of stylized perspective views based on first stylized imageand the second stylized image.

In Example 18, the subject matter of Examples 15-17 includes, generatinga second filtered image based on application of a second guided filterto the second stylized image, wherein the generation of the secondstylized image is further based on the second filtered image.

In Example 19, the subject matter of Example 18 includes, generating afirst filtered image based on application of a first guided filter tothe first stylized image, wherein the generation of the first stylizedimage is further based on the first filtered image.

In Example 20, the subject matter of Example 19 includes, wherein: thefirst guided filter includes a first guided sharpening filter, the firstguided sharpening filter to sharpen the first stylized image based on afirst plurality of edges within the first image; and the second guidedfilter includes a second guided sharpening filter, the second guidedsharpening filter to sharpen the second stylized image based on a secondplurality of edges within the second image.

In Example 21, the subject matter of Examples 15-20 includes, generatingthe first stylized image based on the first image and the style transfermodel.

In Example 22, the subject matter of Example 21 includes, wherein thestyle transfer model includes a style transfer neural network trainedvia machine learning on a plurality of target style images.

In Example 23, the subject matter of Examples 21-22 includes, receivinga target style image; and identifying a target style based onapplication of a neural style transfer (NST) algorithm to the targetstyle image; and generating the style transfer model based on the targetstyle.

In Example 24, the subject matter of Examples 15-23 includes, generatingthe first disparity map based on the first image and the second image,the first disparity map representing differences in horizontalcoordinates of a first plurality of image points in the first imagerelative to a second plurality of image points in the second image; andgenerating the second disparity map based on the first image and thesecond image, the second disparity map representing differences inhorizontal coordinates of the second plurality of image points in thesecond image relative to the first plurality of image points in thefirst image.

In Example 25, the subject matter of Examples 15-24 includes, generatinga first extrapolated image based on the first stylized image and thefirst disparity map, the first extrapolated image representing a firstsynthesized viewpoint extrapolated to a third viewpoint based on thefirst perspective of the scene associated with the first image; andgenerating a second extrapolated image based on the second stylizedimage and the second disparity map, the second extrapolated imagerepresenting a second synthesized viewpoint extrapolated to a fourthviewpoint based on the second perspective of the scene associated withthe second image.

In Example 26, the subject matter of Examples 15-25 includes, generatinga first stylized viewpoint image based on re-projecting the firststylized image from a first desired output viewpoint; and generating asecond stylized viewpoint image based on re-projecting the secondstylized image from a second desired output viewpoint.

In Example 27, the subject matter of Example 26 includes, wherein: thefirst desired output viewpoint is based on a first device viewpointassociated with a device multiview display; and the second desiredoutput viewpoint is based on a second device viewpoint associated with adevice multiview display.

In Example 28, the subject matter of Examples 26-27 includes, generatinga third stylized viewpoint image based on re-projecting the thirdstylized image from a third desired output viewpoint; and generating afourth stylized viewpoint image based on re-projecting the fourthstylized image from a fourth desired output viewpoint.

Thus, there have been described examples and embodiments of a multiviewstyle transfer system and method to display a first stylized image and asecond stylized image on a multiview display. It should be understoodthat the above-described examples are merely illustrative of some of themany specific examples that represent the principles described herein.Clearly, those skilled in the art can readily devise numerous otherarrangements without departing from the scope as defined by thefollowing claims.

What is claimed is:
 1. A multiview style transfer system comprising:processing circuitry; and a memory that includes instructions, theinstructions, when executed by the processing circuitry, cause theprocessing circuitry to: receive a first image representative of a firstperspective of a scene, a first disparity map corresponding to the firstimage, and a second image representative of a second perspective of thescene; generate a first stylized image representative of a styletransfer model applied to the first image; generate a stylized shiftedimage based on the first stylized image and the first disparity map, thestylized shifted image including a first shift of the first stylizedimage to the second perspective; and generate a second stylized imagebased on a guided filter of the stylized shifted image and the secondimage, the guided filter to process edge characteristics in the secondstylized image based on the second image.
 2. The multiview styletransfer system of claim 1, further including a multiview display,wherein the instructions further cause the processing circuitry to:generate the first stylized image based on the first disparity map;generate the second stylized image based on a second disparity mapcorresponding to the second image, wherein the first stylized image andthe second stylized image are configured to be concurrently rendered bya multiview display; and display the first stylized image and the secondstylized image on the multiview display.
 3. The multiview style transfersystem of claim 2, wherein the instructions further cause the processingcircuitry to: synthesize a plurality of stylized perspective views basedon the first stylized image and the second stylized image; and displaythe plurality of stylized perspective views on the multiview display. 4.The multiview style transfer system of claim 1, wherein the instructionsfurther cause the processing circuitry to generate the second stylizedimage as a second filtered image based on application of a second guidedfilter to the second stylized image, wherein the generation of thesecond filtered image is further based on the second image.
 5. Themultiview style transfer system of claim 4, wherein the instructionsfurther cause the processing circuitry to generate the first stylizedimage as a first filtered image based on application of a first guidedfilter to the first stylized image, wherein the generation of the firstfiltered image is further based on the first image.
 6. The multiviewstyle transfer system of claim 5, wherein: the first guided filterincludes a first guided sharpening filter, the first guided sharpeningfilter to sharpen the first stylized image based on a first plurality ofedges within the first image; and the second guided filter includes asecond guided sharpening filter, the second guided sharpening filter tosharpen the second stylized image based on a second plurality of edgeswithin the second image.
 7. The multiview style transfer system of claim1, wherein the instructions further cause the processing circuitry togenerate the first stylized image based on the first image and the styletransfer model.
 8. The multiview style transfer system of claim 7,wherein the style transfer model includes a style transfer neuralnetwork trained via machine learning on a plurality of target styleimages.
 9. The multiview style transfer system of claim 7, wherein theinstructions further cause the processing circuitry to: receive a targetstyle image; identify a target style based on application of a neuralstyle transfer (NST) algorithm to the target style image; and generatethe style transfer model based on the target style.
 10. The multiviewstyle transfer system of claim 1, wherein the first disparity map isgenerated based on the first image and the second image, the firstdisparity map representing differences in horizontal coordinates of afirst plurality of image points in the first image relative to a secondplurality of image points in the second image; and wherein a seconddisparity map is generated based on the first image and the secondimage, the second disparity map representing differences in horizontalcoordinates of the second plurality of image points in the second imagerelative to the first plurality of image points in the first image. 11.The multiview style transfer system of claim 2, wherein the instructionsfurther cause the processing circuitry to: generate a first extrapolatedimage based on the first stylized image and the first disparity map, thefirst extrapolated image representing a first synthesized viewpointextrapolated to a third viewpoint based on the first perspective of thescene associated with the first image; generate a second extrapolatedimage based on the second stylized image and the second disparity map,the second extrapolated image representing a second synthesizedviewpoint extrapolated to a fourth viewpoint based on the secondperspective of the scene associated with the second image; and displaythe first extrapolated image and the second extrapolated image on themultiview display.
 12. The multiview style transfer system of claim 2,wherein the instructions further cause the processing circuitry to:generate a first stylized viewpoint image based on re-projecting thefirst stylized image from a first desired output viewpoint; generate asecond stylized viewpoint image based on re-projecting the secondstylized image from a second desired output viewpoint; and display thefirst stylized viewpoint image and the second stylized viewpoint imageon the multiview display.
 13. The multiview style transfer system ofclaim 12, wherein: the first desired output viewpoint is based on afirst device viewpoint associated with a device multiview display; andthe second desired output viewpoint is based on a second deviceviewpoint associated with the device multiview display.
 14. Themultiview style transfer system of claim 12, wherein the instructionsfurther cause the processing circuitry to: generate a third stylizedviewpoint image based on re-projecting a third stylized image from athird desired output viewpoint; generate a fourth stylized viewpointimage based on re-projecting a fourth stylized image from a fourthdesired output viewpoint; and display the third stylized viewpoint imageand the fourth stylized viewpoint image on the multiview display.
 15. Amultiview style transfer method comprising: receiving a first imagerepresentative of a first perspective of a scene, a first disparity mapcorresponding to the first image, and a second image representative of asecond perspective of the scene; generating a first stylized imagerepresentative of a style transfer model applied to the first image;generating a stylized shifted image based on the first stylized imageand the first disparity map, the stylized shifted image including afirst shift of the first stylized image to the second perspective; andgenerating a second stylized image based on a guided filter of thestylized shifted image and the second image, the guided filter toprocess edge characteristics in the second stylized image based on thesecond image.
 16. The multiview style transfer method of claim 15,further comprising: generating a first stylized image based on the firstdisparity map; and generating a second stylized image based on a seconddisparity map corresponding to the second image, wherein the firststylized image and the second stylized image are configured to berendered by a multiview display.
 17. The multiview style transfer methodof claim 16, further comprising synthesizing a plurality of stylizedperspective views based on the first stylized image and the secondstylized image.
 18. The multiview style transfer method of claim 16,further comprising generating the second stylized image as a secondfiltered image based on application of a second guided filter to thesecond stylized image, wherein the generation of the second filteredimage is further based on the second image.
 19. The multiview styletransfer method of claim 18, further comprising generating the firststylized image as a first filtered image based on application of a firstguided filter to the first stylized image, wherein the generation of thefirst filtered image is further based on the first image.
 20. Themultiview style transfer method of claim 19, wherein: the first guidedfilter includes a first guided sharpening filter, the first guidedsharpening filter to sharpen the first stylized image based on a firstplurality of edges within the first image; and the second guided filterincludes a second guided sharpening filter, the second guided sharpeningfilter to sharpen the second stylized image based on a second pluralityof edges within the second image.
 21. The multiview style transfermethod of claim 15, further comprising generating the first stylizedimage based on the first image and the style transfer model.
 22. Themultiview style transfer method of claim 21, wherein the style transfermodel includes a style transfer neural network trained via machinelearning on a plurality of target style images.
 23. The multiview styletransfer method of claim 21, further comprising: receiving a targetstyle image; identifying a target style based on application of a neuralstyle transfer (NST) algorithm to the target style image; and generatingthe style transfer model based on the target style.
 24. The multiviewstyle transfer method of claim 15, wherein the first disparity map isgenerated based on the first image and the second image, the firstdisparity map representing differences in horizontal coordinates of afirst plurality of image points in the first image relative to a secondplurality of image points in the second image; and wherein a seconddisparity map is generated based on the first image and the secondimage, the second disparity map representing differences in horizontalcoordinates of the second plurality of image points in the second imagerelative to the first plurality of image points in the first image. 25.The multiview style transfer method of claim 16, further comprising:generating a first extrapolated image based on the first stylized imageand the first disparity map, the first extrapolated image representing afirst synthesized viewpoint extrapolated to a third viewpoint based onthe first perspective of the scene associated with the first image; andgenerating a second extrapolated image based on the second stylizedimage and the second disparity map, the second extrapolated imagerepresenting a second synthesized viewpoint extrapolated to a fourthviewpoint based on the second perspective of the scene associated withthe second image.
 26. The multiview style transfer method of claim 16,further comprising: generating a first stylized viewpoint image based onre-projecting the first stylized image from a first desired outputviewpoint; and generating a second stylized viewpoint image based onre-projecting the second stylized image from a second desired outputviewpoint.
 27. The multiview style transfer method of claim 26, wherein:the first desired output viewpoint is based on a first device viewpointassociated with a device multiview display; and the second desiredoutput viewpoint is based on a second device viewpoint associated withthe device multiview display.
 28. The multiview style transfer method ofclaim 26, further comprising: generating a third stylized viewpointimage based on re-projecting the third stylized image from a thirddesired output viewpoint; and generating a fourth stylized viewpointimage based on re-projecting the fourth stylized image from a fourthdesired output viewpoint.